Learning to Skim Text

Background. The last few years have seen much success of applying deep networks in many important applications in Natural Language Processing: sentiment analysis, document classification, machine translation, conversational/dialogue modeling, automatic Q&A. An important trait of all of these models is that they read all the text available to them. While it is essential for certain applications, such as machine translation, this trait also makes it difficult to apply these models to applications that have long input text, such as document classification or automatic Q&A. Aim. We consider the problem of long document understanding and propose a modification to the basic neural architectures that allow them to read input text non-sequentially. The main benefit of this approach is faster inference because it skips irrelevant information. An unexpected benefit this approach is that it sometimes helps the models generalize better. Data. The tasks under test include synthetic number prediction (synthetic data), sentiment classification (Rotten Tomatoes and IMDB), news topic classification (AG) and reading comprehension (Children’s Book Test). Those are representative tasks in text reading involving different sizes of datasets and various levels of text processing. Methods. In our approach, the model is a recurrent network, which learns to predict the number of jumping steps after it reads an input token. The model is therefore not fully differentiable, but it can be trained by a policy gradient algorithm called REINFORCE. The reward of the recurrent network is to optimize the accuracy of the model on the training dataset. Anticipated results. The comparison is between the vanilla LSTM and our model. In a nutshell, we anticipate that, while achieving the same testing accuracy, our model is much faster than the baseline LSTM model, as we are able to skip a bunch of text. Conclusions. The model we develop can indeed learn how to “jump” while processing text, which is faster than most of the existing methods.